Projection type display device and projection control method

ABSTRACT

Provided are a projection type display device and a projection control method capable of accurately performing information presentation to an operator of a vehicle even in an image light shielded state where image light to be projected onto a projection surface is shielded by an object. An HUD 10 that is mounted in a construction machine 100 having a windshield 5 notifies, in a case where it is detected that image light to be projected onto the windshield 5 is in an image light shielded state where at least a part of the image light is shielded by an object, such as part of the body of an operator of a vehicle, the presence of the state.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of PCT International Application No.PCT/JP2016/057325 filed on Mar. 9, 2016, which claims priority under 35U.S.C § 119(a) to Japanese Patent Application No. 2015-184047 filed onSep. 17, 2015. Each of the above application(s) is hereby expresslyincorporated by reference, in its entirety, into the presentapplication.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a projection type display device and aprojection control method.

2. Description of the Related Art

A vehicle head-up display (HUD) that projects, using a windshield of avehicle such as an automobile or a combiner disposed in the vicinity ofthe windshield as a screen, light onto the screen to display an image isknown (for example, refer to JP2013-67209A). According to the HUD, auser can set an image based on the light projected from the HUD as areal image on the screen, or can set the image as a virtual image infront of the screen, so that a driver can visually recognize the image.

SUMMARY OF THE INVENTION

In a construction machine, movement of a line of sight of an operator isfrequently performed, particularly, in a longitudinal direction,differently from a vehicle of which main purpose is transportation, suchas an automobile. Further, a movement range of the line of sight of theoperator in the longitudinal direction is wide differently from thevehicle of which main purpose is transportation. In addition, in theconstruction machine, the line of sight of the operator moves inaccordance with movement of a power shovel and/or a bucket that is anoperation target. In consideration of these points, in a constructionmachine with a windshield in front of an operator's seat, it ispreferable that a virtual image and a real image are visuallyrecognizable over a wide range of the windshield.

In this way, in a case where image light is projected over a wide rangeof a windshield, there is a high possibility that a part of theoperator's body (for example, hands and/or head, or the like) shieldsthe image light. Information to be presented to the operator using anHUD includes always-existing information that needs to be displayed atall times and not-always-existing information that need to be displayedunder a specific condition such as a warning display.

It is assumed that image light projected over a range where thenot-always-existing information is displayed is shielded by a part ofthe operator's body. In this case, since it cannot be said thatinformation is always displayed over the range, it is difficult for theoperator to recognize whether the image light is shielded. In thisstate, in a case where image light for displaying thenot-always-existing information is projected, the operator cannot checkthe not-always-existing information.

Even in a range where the always-existing information is displayed, in acase where visibility of information itself is lowered, for example, ina case where the surroundings are bright, it is difficult for theoperator to notice that the always-existing information is shielded by apart of the body.

Here, the construction machine is described as an example, but a statewhere image light is shielded by a part of the body of a user, or thelike may similarly occur in an HUD mounted in either a working machinesuch as a construction machine or an agricultural machine such as atractor or a vehicle of which main purpose is transportation. Further,even in such a vehicle, under the environment where a user concentrateson driving, it is difficult for the user to notice a state where imagelight is shielded by a part of the user's body, or the like.

JP2013-67209A discloses a vehicle in which an HUD is provided in aceiling portion that is a position where there is a high possibilitythat image light is shielded by the body of a user. However, thistechnique does not consider the problem that it is difficult for theuser to recognize a state where the image light is shielded.

The invention has been made in consideration of the above-mentionedproblems, and an object of the invention is to provide a projection typedisplay device and a projection control method capable of accuratelypresenting information to an operator of a vehicle even in a case whereimage light projected onto a projection surface is shielded by anobject.

According to an aspect of the invention, there is provided a projectiontype display device comprising: a projection display unit that projectsimage light, which is spatially modulated by a light modulation elementthat spatially modulates light emitted from a light source on the basisof image information, onto a projection surface mounted in a vehicle; adetection unit that is capable of detecting an image light shieldedstate where at least a part of the image light is shielded by part of abody of an operator of the vehicle; and a control unit that performs anotification process in a case where the image light shielded state isdetected by the detection unit.

According to another aspect of the invention, there is provided aprojection control method for projecting image light spatially modulatedby a light modulation element that spatially modulates light emittedfrom a light source on the basis of image information onto a projectionsurface mounted in a vehicle, comprising: a detection step of detectingan image light shielded state where at least a part of the image lightis shielded by part of a body of an operator of the vehicle; and acontrol step of performing a notification process in a case where theimage light shielded state is detected in the detection step.

According to the invention, it is possible to provide a projection typedisplay device and a projection control method capable of accuratelypresenting information to an operator of a vehicle even in a case whereimage light projected onto a projection surface is shielded by anobject.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing a schematic configuration of aconstruction machine 100 provided with an HUD 10 that is an embodimentof a projection type display device of the invention.

FIG. 2 is a diagram showing an example of a configuration in anoperator's cab in the construction machine 100 shown in FIG. 1.

FIG. 3 is a schematic diagram showing an internal configuration of aunit 2 that forms the HUD 10 shown in FIG. 1.

FIG. 4 is a schematic diagram showing an internal configuration of aunit 3 that forms the HUD 10 shown in FIG. 1.

FIG. 5 is a schematic diagram showing an internal configuration of aunit 4 that forms the HUD 10 shown in FIG. 1.

FIG. 6 is a diagram showing a functional block of a main controller 70that forms the unit 2 shown in FIG. 3.

FIG. 7 is a diagram schematically showing an image light shielded state.

FIGS. 8A and 8B are schematic diagrams for illustrating a firstnotification process.

FIGS. 9A and 9B are schematic diagrams for illustrating a firstnotification process.

FIGS. 10A and 10B are schematic diagrams for illustrating a thirdnotification process.

FIG. 11 is a flowchart for illustrating an operation of the HUD 10 shownin FIG. 1.

FIG. 12 is a diagram showing a functional block of a main controller 70Athat is a modification example of the main controller 70 shown in FIG.3.

FIGS. 13A and 13B are schematic diagrams for illustrating functions ofthe main controller 70A.

FIG. 14 is a flowchart for illustrating an operation of an HUD 10 havingthe main controller 70A shown in FIG. 12.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the invention will be described withreference to the accompanying drawings.

FIG. 1 is a schematic diagram showing a schematic configuration of aconstruction machine 100 provided with an HUD 10 that is an embodimentof a projection type display device of the invention.

The HUD 10 shown in FIG. 1 is mounted in a working machine such as aconstruction machine and an agricultural machine, and a vehicle such asan automobile, a train, an airplane, or a ship.

The HUD 10 includes a unit 2 that is provided on an upper and rear sideof an operator's seat 1 in an operator's cab, a unit 3 that is providedon a rear side of the operator's seat 1 in the operator's cab, and aunit 4 that is provided on a lower and rear side of the operator's seat1 in the operator's cab. The operator's seat means an indoor space ofthe construction machine 100.

The units 2 to 4 are provided to be spaced from each other in a gravitydirection (a longitudinal direction in FIG. 1) in the operator's cab ofthe construction machine 100. Each of the units 2 to 4 projects imagelight under the condition that a virtual image is visually recognizablein front of a windshield 5 of the construction machine 100.

An operator of the construction machine 100 can visually recognizeinformation on a picture, characters, or the like for assisting anoperation of the construction machine 100 by viewing image light that isprojected onto the windshield 5 and is reflected therefrom. Further, thewindshield 5 has a function of reflecting image light projected fromeach of the units 2 to 4 and simultaneously transmitting light from theoutside (an outside world). Thus, the operator can visually recognize avirtual image based on the image light projected from each of the units2 to 4 in a state where the virtual image is superimposed on a scene ofthe outside world.

In the HUD 10, since the units 2 to 4 are provided to be spaced fromeach other in the gravity direction in the operator's cab of theconstruction machine 100, it is possible to present a virtual image tothe operator over a wide range of the windshield 5.

An imaging unit 6 is provided in the operator's cab above the operatorof the construction machine 100. The imaging unit 6 captures a range inthe operator's cab that covers the entirety of an optical path of imagelight projected from each of the units 2 to 4.

The imaging unit 6 employs a stereo camera that uses two imagingelements, for example. The imaging unit 6 is connected to the unit 2that forms the HUD 10 in a wired or wireless manner, and transmits dataon a captured image obtained by imaging the inside of the operator's cabto the unit 2.

On a dashboard 7 of the construction machine 100, a display device 8that is formed by a liquid crystal display or the like is provided. Thedisplay device 8 is connected to the unit 2 that forms the HUD 10 in awired or wireless manner, and displays information under the control ofthe unit 2.

A sound output device 9 that is formed by a speaker or the like isprovided on a ceiling of the operator's cab of the construction machine100. The sound output device 9 is connected to the unit 2 that forms theHUD 10 in a wired or wireless manner, and outputs sound under thecontrol of the unit 2.

FIG. 2 is a schematic diagram showing an example of a configuration inthe operator's cab in the construction machine 100 shown in FIG. 1. FIG.2 shows a front view in a state where the windshield 5 is seen from theoperator's seat 1.

The construction machine 100 is a hydraulic shovel that includes an arm21 and a bucket 22 in a front center of the machine.

The operator's cab is surrounded by transparent windows such as thewindshield 5 that is a front window, a right window 23, a left window24, and the like.

In the operator's cab, a left operating lever 25 for operating bendingand stretching of the arm 21, a right operating lever 26 for operatingdigging and opening of the bucket 22, and the like are provided aroundthe operator's seat 1.

Three projection ranges of a first projection range 5A, a secondprojection range 5B, and a third projection range 5C are sequentiallyallocated onto the windshield 5, and the projection ranges are arrangedin the gravity direction (longitudinal direction). Here, a rangeobtained by combining the three projection ranges on the windshield 5forms a projection surface mounted in the construction machine 100.

The first projection range 5A is a region where image light projectedfrom the unit 2 is projected, which reflects the image light andsimultaneously transmits light from the outside (outside world).

The second projection range 5B is a region where image light projectedfrom the unit 3 is projected, which reflects the image light andsimultaneously transmits light from the outside (outside world).

The third projection range 5C is a region where image light projectedfrom the unit 4 is projected, which reflects the image light andsimultaneously transmits light from the outside (outside world).

FIG. 3 is a schematic diagram showing an internal configuration of theunit 2 that forms the HUD 10 shown in FIG. 1.

The unit 2 includes a light source unit 40, a driving unit 45, aprojection optical system 46, a diffuser plate 47, a reflecting mirror48, a magnifying glass 49, a system controller 60 that controls a lightsource controller 40A and the driving unit 45, and a main controller 70.

The light source unit 40 includes the light source controller 40A, an Rlight source 41 r that is a red light source that emits red light, a Glight source 41 g that is a green light source that emits green light, aB light source 41 b that is a blue light source that emits blue light, adichroic prism 43, a collimator lens 42 r that is provided between the Rlight source 41 r and the dichroic prism 43, a collimator lens 42 g thatis provided between the G light source 41 g and the dichroic prism 43, acollimator lens 42 b that is provided between the B light source 41 band the dichroic prism 43, and a light modulation element 44.

The dichroic prism 43 is an optical member for guiding light emittedfrom each of the R light source 41 r, the G light source 41 g, and the Blight source 41 b to the same optical path. That is, the dichroic prism43 transmits red light that is collimated by the collimator lens 42 r tobe emitted to the light modulation element 44. Further, the dichroicprism 43 reflects green light that is collimated by the collimator lens42 g to be emitted to the light modulation element 44. Further, thedichroic prism 43 reflects blue light that is collimated by thecollimator lens 42 b to be emitted to the light modulation element 44.An optical member having such a function is not limited to a dichroicprism. For example, a cross dichroic mirror may be used.

The R light source 41 r, the G light source 41 g, and the B light source41 b respectively employ a light emitting element such as laser or alight emitting diode (LED). In this embodiment, an example in which theHUD includes three light sources of the R light source 41 r, the G lightsource 41 g, and the B light source 41 b, as the light sources of thelight source unit 40, is shown, but the number of light sources may be1, 2, 4 or more.

The light source controller 40A sets the amounts of luminescence of theR light source 41 r, the G light source 41 g, and the B light source 41b into predetermined luminescence amount patterns, and performs acontrol for sequentially emitting light from the R light source 41 r,the G light source 41 g, and the B light source 41 b according to theluminescence amount patterns.

The light modulation element 44 spatially modulates light emitted fromthe dichroic prism 43 on the basis of projection image data that isimage information, and emits light (red color image light, blue colorimage light, and green color image light) based on the projection imagedata to the projection optical system 46.

The light modulation element 44 may employ, for example, a liquidcrystal on silicon (LCOS), a digital micro-mirror device (DMD), a microelectro mechanical systems (MEMS) element, a liquid crystal displaydevice, or the like.

The driving unit 45 drives the light modulation element 44 according toprojection image data input from the system controller 60, so that light(red color image light, blue color image light, and green color imagelight) based on the projection image data is emitted to the projectionoptical system 46.

The projection optical system 46 is an optical system for projectinglight emitted from the light modulation element 44 of the light sourceunit 40 onto the diffuser plate 47. The optical system is not limited toa lens, and may employ a scanner. For example, the diffuser plate 47 maydiffuse light emitted from a scanning-type scanner to become a planelight source.

The reflecting mirror 48 reflects light diffused by the diffuser plate47 toward the magnifying glass 49.

The magnifying glass 49 magnifies an image based on light reflected bythe reflecting mirror 48, and projects the magnified image onto thefirst projection range 5A of the windshield 5.

In a case where an image light projection command is received from themain controller 70, the system controller 60 projects image light basedon projection image data transmitted from the main controller 70 ontothe first projection range 5A.

The main controller 70 generally controls the entirety of the HUD 10,and is able to communicate with each of the units 3 and 4 in a wirelessor wired manner. Further, the main controller 70 is also able tocommunicate with the display device 8 and the sound output device 9 in awireless or wired manner. Detailed functions of the main controller 70will be described later.

FIG. 4 is a schematic diagram showing an internal configuration of theunit 3 that forms the HUD 10 shown in FIG. 1. In FIG. 4, the samecomponents as in FIG. 3 are given the same reference numerals.

The unit 3 has a configuration in which the main controller 70 isremoved in the unit 2 shown in FIG. 3 and the system controller 60 ismodified into a system controller 61.

The system controller 61 of the unit 3 controls the driving unit 45 andthe light source controller 40A in the unit 3, so that image light basedon projection image data transmitted from the main controller 70 isprojected onto the second projection range 5B.

The system controller 61 is able to communicate with the main controller70 of the unit 2. In a case where an image light projection command isreceived from the main controller 70, the system controller 61 projectsimage light based on projection image data transmitted from the maincontroller 70 onto the second projection range 5B.

FIG. 5 is a schematic diagram showing an internal configuration of theunit 4 that forms the HUD 10 shown in FIG. 1. In FIG. 5, the samecomponents as in FIG. 3 are given the same reference numerals.

The unit 4 has a configuration in which the main controller 70 in theunit 2 shown in FIG. 3 is removed and the system controller 60 ismodified into a system controller 62.

The system controller 62 of the unit 4 controls the driving unit 45 andthe light source controller 40A in the unit 4, so that image light basedon projection image data transmitted from the main controller 70 isprojected onto the third projection range 5C.

The system controller 62 is able to communicate with the main controller70 of the unit 2. In a case where an image light projection command isreceived from the main controller 70, the system controller 62 projectsimage light based on projection image data onto the third projectionrange 5C.

The light source unit 40, the projection optical system 46, the diffuserplate 47, the reflecting minor 48, and the magnifying glass 49 in theunit 2 form a projection unit that projects image light based onprojection image data onto the first projection range 5A.

The light source unit 40, in the unit 3, the projection optical system46, the diffuser plate 47, the reflecting mirror 48, and the magnifyingglass 49 form a projection unit that projects image light based on theprojection image data onto the second projection range 5B.

The light source unit 40, the projection optical system 46, the diffuserplate 47, the reflecting mirror 48, and the magnifying glass 49 in theunit 4 form a projection unit that projects image light based on theprojection image data onto the third projection range 5C.

The three projection units form a projection display unit of the HUD 10.The projection display unit of the HUD 10 is disposed at a positiondistant from the windshield 5 that is the projection surface withreference to the operator's seat 1, in the operator's cab of theconstruction machine 100.

FIG. 6 is a diagram showing a functional block of the main controller 70that forms the unit 2 shown in FIG. 3. As shown in FIG. 6, the maincontroller 70 includes a captured image data acquisition unit 71, adetection unit 72, and a notification processing unit 73.

The captured image data acquisition unit 71 acquires data on a capturedimage obtained using the imaging unit 6 shown in FIG. 1, and inputs theresult to the detection unit 72.

The detection unit 72 detects a state where at least a part of imagelight projected onto the projection surface (all the ranges includingthe first projection range 5A, the second projection range 5B, and thethird projection range 5C) of the construction machine 100 from theunits 2 to 4 is shielded by an object (for example, an operator)(hereinafter, referred to as an image light shielded state).

FIG. 7 is a diagram schematically showing the image light shieldedstate.

FIG. 7 shows a range 100 where image light projected from the unit 2passes, a range 300 where image light projected from the unit 3 passes,and a range 400 where image light projected from the unit 4 passes. Inthe example of FIG. 7, an object overlaps over a part of the range 300and a part of the range 400, and this overlapping state corresponds tothe image light shielded state.

In order to detect the image light shielded state. the detection unit 72uses captured image data acquired by the captured image data acquisitionunit 71.

From the captured image data, a three-dimensional shape of each of therange 100, the range 300, and the range 400 shown in FIG. 7, in theoperator's cab, can be known. Further, in a normal operating state ofthe HUD 10, it is assumed that an object is not present on each of therange 100, the range 300, and the range 400.

Thus, the detection unit 72 detects the image light shielded stateaccording to whether an object image having a certain size or greater isincluded in at least one of a portion corresponding to the range 100, aportion corresponding to the range 300, or a portion corresponding tothe range 400, in the captured image data.

In a case where the image light shielded state is detected by theabove-described method, the detection unit 72 outputs a detection signalto the notification processing unit 73.

The detection signal includes information on a range where an objectimage is detected in the captured image data (in the example of FIG. 7,the range 300 and the range 400), and information on the position of theobject image in the range.

The notification processing unit 73 performs a notification process in acase where the detection signal is input from the detection unit 72. Thenotification processing unit 73 forms a control unit.

The notification process performed by the notification processing unit73 employs a first notification process to a fifth notification processto be described hereinafter, for example, but are not limited to thesemethods.

(First Notification Process)

In a case where a detection signal is input from the detection unit 72,the notification processing unit 73 changes (updates) projection imagedata into projection image data to which notification informationindicating the image light shielded state is added, and projects imagelight based on the projection image data onto the windshield 5 to notifythe presence of the image light shielded state.

The first notification process will be specifically described withreference to FIGS. 8A and 8B and FIGS. 9A and 9B.

FIG. 8A is a diagram showing projection image data 200 that is a sourceof image light to be projected onto the windshield 5.

Data 200A in the projection image data 200 represents data correspondingto image light projected onto the first projection range 5A from theunit 2. Data 200B represents data corresponding to image light projectedonto the second projection range 5B from the unit 3. Data 200Crepresents data corresponding to image light projected onto the thirdprojection range 5C from the unit 4.

In the example of FIG. 8A, the data 200A includes an icon 201. The data200B includes an icon 202. The icon 201 and the icon 202 representinformation for performing information presentation to an operator, suchas information for assisting work (construction design information,warning information, or the like), a traveling speed of the constructionmachine 100, or fuel information of the construction machine 100.

FIG. 8B is a diagram showing a state where the image light based on theprojection image data 200 shown in FIG. 8A is projected onto thewindshield 5. As shown in FIGS. 8A and 8B, in front of the projectionsurface of the windshield 5, a virtual image 211 corresponding to theicon 201 is displayed.

As shown in FIG. 7, FIGS. 8A and 8B show a case where an object ispresent over a part of the range 300 and a part of the range 400, inwhich a range 214 (hereinafter, referred to as a missed range) whereimage light shielded by the object is to be originally projected isshown on the projection surface. The missed range 214 corresponds to apart of the image light based on the projection image data 200, shieldedby the object, where the part of the image light shielded by the objectis also referred as the “shielded part of the image light”. A rangeother than the missed range 214 on the projection surface corresponds toa part of the image light based on the projection image data 200 otherthan the part shielded by the object.

The notification processing unit 73 determines the position of themissed range 214 on the projection surface (all the ranges obtained bycombining the first projection range 5A, the second projection range 5B,and the third projection range 5C) shown in FIG. 8B, on the basis ofinformation on the range where the object is present, included in thedetection signal input from the detection unit 72 and information on theposition of the object in the range. The position of the missed range214 means coordinates data in the periphery of the missed range 214.

Further, the notification processing unit 73 generates projection imagedata 200 a obtained by changing data corresponding to a range other thanthe missed range 214 on the projection surface into data includingnotification information (for example, an icon, characters, or the like)for notification of the image light shielded state, in the projectionimage data 200, and transmits data 200A, data 200B, and data 200C of theprojection image data 200 a to the system controllers 60 to 62.

FIG. 9A is a diagram showing the projection image data 200 a. Theprojection image data 200 a is obtained by adding character information203 that is notification information below the icon 201 with respect tothe projection image data 200.

The system controller 60 controls the driving unit 45 and the lightsource controller 40A in the unit 2, so that image light based on thedata 200A of the projection image data 200 a is projected onto the firstprojection range 5A of the windshield 5. Further, the system controller61 in the unit 3 controls the driving unit 45 and the light sourcecontroller 40A in the unit 3, so that image light based on the data 200Bof the projection image data 200 a is projected onto the secondprojection range 5B of the windshield 5. Further, the system controller62 in the unit 4 controls the driving unit 45 and the light sourcecontroller 40A in the unit 4, so that image light based on the data 200Cof the projection image data 200 a is projected onto the thirdprojection range 5C of the windshield 5.

FIG. 9B is a diagram showing a state where the image light based on theprojection image data 200 a shown in FIG. 9A is projected onto thewindshield 5. FIG. 9B shows a virtual image 215 corresponding to thecharacter information 203 included in the projection image data 200 a.The missed range 214 shown in FIGS. 8B and 9B corresponds to an image,and is not actually seen to an operator.

The operator views the virtual image 215 shown in FIG. 9B to therebyrecognize the image light shielded state.

Even in a state where image light is shielded by a part of the body ofthe operator through the first notification process, the operator caneasily recognize the presence of the state. Thus, the operator can movethe operator's body to take an action for prohibiting the image lightshielded state, and thus, it is possible to accurately provide necessaryinformation to the operator.

(Second Notification Process)

The notification processing unit 73 notifies the image light shieldedstate using sound.

Specifically, the notification processing unit 73 outputs soundindicating, for example, “there is a possibility that a part of displayinformation is shielded by an object”, or the like through the soundoutput device 9. The notification processing unit 73 may notify thepresence of the image light shielded state using a buzzer sound or amelody sound, instead of using the sound message.

Even in a case where image light is shielded by a part of the operator'sbody, through the second notification process, the operator can easilyrecognize the state using sound.

By combining the first notification process and the second notificationprocess, it is possible to cause the operator to accurately recognizethe image light shielded state.

(Third Notification Process)

In a case where a detection signal is input from the detection unit 72,the notification processing unit 73 extracts, from data corresponding toa part of image light, shielded by an object, in projection image data,shielding information (important information that may cause obstacles inworking and driving, which does not include information that does notinterference with work, such as time information) to be presented to anoperator of the construction machine 100, included in the data, andperforms of notification of the shielding information.

The third notification process will be described with reference to FIGS.8A and 8B and FIGS. 10A and 10B.

The notification processing unit 73 determines the position of themissed range 214 on the projection surface (all the ranges obtained bycombining the first projection range 5A, the second projection range 5B,and the third projection range 5C) shown in FIG. 8B, on the basis ofinformation on a range where an object is present, included in thedetection signal input from the detection unit 72, and information onthe position of the object in the range.

Further, the notification processing unit 73 determines whethershielding information to be presented to the operator, such asinformation for assisting work, a traveling speed of the constructionmachine 100, or fuel information of the construction machine 100, isincluded in the data of the part corresponding to the missed range 214in the projection image data 200.

In a case where it is determined that the shielding information to bepresented to the operator is included in the data of the partcorresponding to the missed range 214 in the projection image data 200,the notification processing unit 73 generates projection image data 200b obtained by adding the shielding information included in the data ofthe part corresponding to the missed range 214 to data corresponding toa part excluding the missed range 214 in the projection image data 200,and transmits data 200A, data 200B, and data 200C of the projectionimage data 200 b to the system controllers 60 to 62.

FIG. 10A is a diagram showing the projection image data 200 b. FIG. 10Bis a diagram showing a state where image light based on the projectionimage data 200 b shown in FIG. 10A is projected onto the windshield 5.

As shown in FIG. 10A, the icon 202 disposed at a position indicated by abroken line in the projection image data 200 is moved to below the icon201 in the projection image data 200 b. This is because the icon 202 isincluded in the data corresponding to the missed range 214 in theprojection image data 200.

In this way, the notification processing unit 73 generates theprojection image data 200 b obtained by removing the icon 201 that isshielding information in the projection image data 200, and by addingthe icon 202 to a part other than the data corresponding to the missedrange 214 in the projection image data 200.

In a case where the projection image data 200 b is generated, the systemcontroller 60 controls the driving unit 45 and the light sourcecontroller 40A in the unit 2, so that image light based on the data 200Aof the projection image data 200 b is projected onto the firstprojection range 5A of the windshield 5. Further, the system controller61 in the unit 3 controls the driving unit 45 and the light sourcecontroller 40A in the unit 3, so that image light based on the data 200Bof the projection image data 200 b is projected onto the secondprojection range 5B of the windshield 5. Further, the system controller62 in the unit 4 controls the driving unit 45 and the light sourcecontroller 40A in the unit 4, so that image light based on the data 200Cof the projection image data 200 b is projected onto the thirdprojection range 5C of the windshield 5.

Thus, the state of FIG. 8B transitions to the state of FIG. 10B. FIG.10B shows a virtual image 216 corresponding to the icon 202 included inthe projection image data 200 b. The missed range 214 shown in FIG. 10Bcorresponds to an image, and is not actually seen to an operator.

According to the third notification process, even in a case whereinformation (icon 202 in FIG. 10A) to be presented to the operator isshielded by a part of the operator's body, or the like, since thevirtual image 216 corresponding to the icon 202 is moved to a positionwhere the virtual image 216 is visually recognized, the operator caneasily recognize the shielding information. Accordingly, it is possibleto accurately perform working assistance.

Fourth Notification Process

The notification processing unit 73 notifies the shielding informationdescribed in the third notification process using sound.

Specifically, in a case where it is determined that there is shieldinginformation, the notification processing unit 73 outputs a message,boozer sound, melody sound, or the like corresponding to the shieldinginformation, from the sound output device 9.

For example, in a case where the shielding information is an icon forwarning fuel shortage, the notification processing unit 73 outputs amessage “Fuel is running low. Please refill.” through the sound outputdevice 9.

According to the fourth notification process, even in a case whereinformation to be presented to an operator is shielded by a part of theoperator, or the like, it is possible to notify shielding informationusing sound, and thus, the operator can easily recognize the shieldinginformation. Thus, it is possible to accurately perform workingassistance.

By combining the third notification process and the fourth notificationprocess, it is possible to cause the operator to more accuratelyrecognize the shielding information.

Fifth Notification Process

The notification processing unit 73 performs notification by displayingthe shielding information described in the third notification process onthe display device 8.

Specifically, in a case where it is determined that there is shieldinginformation, the notification processing unit 73 displays a messagecorresponding to the shielding information (for example, “Fuel isrunning low. Please refill.”) on the display device 8. According to thisconfiguration, even in an environment where sound is difficult to hear,it is possible to easily recognize the shielding information.

FIG. 11 is a flowchart for illustrating an operation of the HUD 10 shownin FIG. 1.

In a case where the HUD 10 is started, the main controller 70 controlsthe system controller 60, the system controller 61, and the systemcontroller 62 to project image light based on projection image data ontothe windshield 5.

In a case where the HUD 10 is started, the imaging unit 6 startsimaging, and transmits data on a captured image obtained through theimaging to the main controller 70 of the unit 2. The captured image dataacquisition unit 71 acquires the captured image data obtained using theimaging unit 6 (step S1).

The detection unit 72 performs a process of detecting whether an imagelight shielded state is present on the basis of the captured image dataacquired using the captured image data acquisition unit 71 andinformation (stored in an internal memory in advance) on a rangeindicating an optical path of image light of each of the units 2 to 4(step S2).

In a case where it is determined that the image light shielded state ispresent (YES in step S3), the detection unit 72 inputs a detectionsignal to the notification processing unit 73.

The notification processing unit 73 performs a notification processusing any one of the above-described methods on the basis of thedetection signal that is input in this way (step S4). After the processof step S4, the procedure returns to step S1, and the above-describedprocesses are repeated.

In a case where it is determined that the image light shielded state isnot present in step S3 (NO in step S3), the notification process is notperformed, and the procedure returns to step S1.

Accordingly, according to the HUD 10 shown in FIG. 1, even in a casewhere it is detected that the image light shielded state is present, itis possible to display notification information indicating the presenceof the image light shielded state using a virtual image, to notify thepresence of the image light shielded state using sound, to display anicon shielded by an object using a virtual image, or to notify a messagecorresponding to the icon shielded by the object using sound and/ordisplay.

Thus, under the environment where an operator concentrates on anoperation, even in a state where image light is shielded by a part ofthe operator's body, or the like, the operator can easily recognize thestate, or can check shielding information. Accordingly, the operator canreceive accurate working assistance, and can perform the operationsmoothly.

A working machine such as a construction machine or an agriculturalmachine has a narrow indoor space. In a case where projection over awide range is performed in such a narrow indoor space, shielding ofimage light due to a part of the operator's body, or the like easilyoccurs. For this reason, the invention is particularly advantageous in acase where the HUD 10 is applied to the working machine such as aconstruction machine or an agricultural machine.

In the above description, a plurality of projection ranges set on thewindshield 5 is arranged in the gravity direction (longitudinaldirection), but the plurality of projection ranges set on the windshield5 may be arranged in a horizontal direction (lateral direction). In thiscase, in the operator's cab of the construction machine 100, aconfiguration in which units that project image light to the respectiveprojection ranges are spaced from each other in the horizontal directionmay be used.

Further, in the above description, the plurality of projection rangesset on the windshield 5 are arranged in one direction, but the pluralityof projection ranges may be arranged in a two-dimensional shape.

Further, in the HUD 10 shown in FIG. 1, a configuration in which theunit 3 and the unit 4 are removed may be used. That is, at least oneprojection unit may be used. In this configuration, since there is apossibility that the image light is shielded, the configuration of themain controller 70 becomes effective.

In the above description, the main controller 70 is provided in the unit2, but a configuration in which a control unit that includes the maincontroller 70 is provided as a separate body and the control unitgenerally controls the system controllers of the units 2 to 4 may beused.

Further, in the above description, all of the units 2 to 4 areconfigured to project image light under the condition that a virtualimage is visually recognizable, but at least one of the units 2 to 4 maybe configured to project image light under the condition that a realimage is visually recognizable.

FIG. 12 is a functional block diagram of a main controller 70A that is amodification example of the main controller 70 of the unit 2 shown inFIG. 3. In FIG. 12, the same components as in FIG. 6 are given the samereference numerals.

As shown in FIG. 12, the main controller 70A includes an external objectdetection unit 74, in addition to the configuration of the maincontroller 70.

In the external object detection unit 74, image feature information onan external object (in this embodiment, a bucket 22) present in a frontcenter of the construction machine 100 is set in advance.

The external object detection unit 74 performs matching based on theimage feature information with respect to a portion corresponding to theprojection surface in the captured image data acquired by the capturedimage data acquisition unit 71, to thereby detect whether the externalobject is present in front of the projection surface. In a case where itis detected that the external object is present, the external objectdetection unit 74 outputs information on the position of the externalobject on the projection surface to the notification processing unit 73.

The notification processing unit 73 determines whether to perform theabove-described notification process on the basis of the information onthe position of the object included in a detection signal input from thedetection unit 72 (information on the position of the object in each ofthe ranges 100, 300, and 400 shown in FIG. 7) and the information on theposition of the external object input from the external object detectionunit 74.

Hereinafter, description will be specifically made with reference toFIGS. 13A and 13B.

FIG. 13A is a diagram showing projection image data 200 c generated bythe main controller 70A. FIG. 13B is a diagram showing a state whereimage light based on the projection image data 200 c shown in FIG. 13Ais projected onto the windshield 5. FIG. 13B shows a missed range 214 ina similar way to FIG. 8B.

In the state shown in FIG. 13B, the external object detection unit 74 ofthe main controller 70A detects, on the basis of data corresponding tothe projection surface in captured image data acquired using the imagingunit 6, the bucket 22 that is an external object from the data. Theexternal object detection unit 74 outputs information on the position ofthe bucket 22 to the notification processing unit 73 as a detectionresult. The information on the position of the bucket 22 is informationon the position of a region 220 having a predetermined size thatsurrounds the bucket 22.

The notification processing unit 73 compares the information on theposition of the missed range 214 on the projection surface input fromthe detection unit 72 with the information on the position of the bucket22 on the projection surface input from the external object detectionunit 74.

In a case where it is determined that the region 220 is outside themissed range 214 through the comparison, the notification processingunit 73 does not perform the above-described notification process in acase where it is determined that the region 220 is outside the missedrange 214, and performs the above-described notification process in acase where it is determined that the region 220 is inside the missedrange 214.

In the construction machine 100, basically, an operator performs anoperation while viewing the vicinity of the bucket 22. Accordingly,concentration on the vicinity of the bucket 22 with respect toinformation to be presented to the operator causes a small movement of asight line, which is preferable.

Thus, the main controller 70A generates the projection image data 200 cso that virtual images 211 and 216 are displayed in the region 220including the bucket 22. In this way, in a case where it is assumed thata virtual image is displayed in the region 220, when image lightprojected onto the region 220 is not shielded, there is no particularinfluence on working assistance.

Accordingly, the notification processing detection unit 73 does notperform the notification process in a case where it is determined thatthe region 220 is outside the missed range 214.

FIG. 14 is a flowchart for illustrating an operation of the HUD 10having the main controller 70A shown in FIG. 12. The flowchart shown inFIG. 14 is the same as the flowchart shown in FIG. 11 except that stepsS5 to S8 are added. In FIG. 14, the same processes as in FIG. 11 aregiven the same reference numerals, and description thereof will not berepeated.

In FIG. 14, in a case where the determination in step S3 is YES, theexternal object detection unit 74 detects an external object in front ofthe projection surface (step S5).

The external object detection unit 74 determines whether the externalobject is detected (step S6). In a case where it is determined that theexternal object is detected (YES in step S6), the external objectdetection unit 74 inputs information on the position of the externalobject to the notification processing unit 73. In a case where it isdetermined that the external object is not detected (NO in step S6), theprocedure returns to step S1.

The notification processing unit 73 determines whether the externalobject detected by the external object detection unit 74 is inside themissed range on the projection surface input from the detection unit 72(step S7).

In the example of FIGS. 13A and 13B, in a case where there is anoverlapping portion between the missed range 214 and the region 220including the external object, the notification processing unit 73determines that the external object is inside the missed range, and in acase where there is no overlapping portion between the missed range 214and the region 220, the notification processing unit 73 determines thatthe external object is outside the missed range.

In a case where the determination of step S7 is YES, the notificationprocessing unit 73 performs the notification process (step S4). Afterthe process of step S4, the procedure returns to step S1, and theabove-described processes are repeated.

In a case where the determination of step S7 is NO, the notificationprocessing unit 73 prohibits the notification process (step S8). Afterthe process of step S8, the procedure returns to step S1, and theabove-described processes are repeated.

As described above, in the HUD 10 of the modification example, in a casewhere the region 220 including the external object and the missed range214 overlap each other, the notification process is performed, and in acase where the region 220 including the external object and the missedrange 214 do not overlap each other, the notification process is notperformed. Thus, it is possible to prevent the notification process frombeing excessively performed, and to efficiently perform an operation.

The position of the bucket 22 is uniquely determined by operationsignals of the left operating lever 25 and the right operating lever 26.Thus, the external object detection unit 74 may detect the position ofthe external object on the basis of the operation signals.

As described above, the following configurations are disclosed in thisspecification.

A disclosed projection type display device includes: a projectiondisplay unit that projects image light spatially modulated by a lightmodulation element that spatially modulates light emitted from a lightsource on the basis of image information onto a projection surfacemounted in a vehicle; a detection unit that is capable of detecting animage light shielded state where at least a part of the image light isshielded by part of a body of an operator of the vehicle; and a controlunit that performs a notification process in a case where the imagelight shielded state is detected by the detection unit.

The disclosed projection type display device is configured so that thenotification process performed by the control unit is a process ofnotifying the presence of the image light shielded state.

The disclosed projection type display device is configured so that thecontrol unit notifies the presence of the image light shielded state bychanging information corresponding to a part of the image lightexcluding the shielded part of the image light to information includingnotification information indicating the presence of the image lightshielded state, in the image information.

The disclosed projection type display device is configured so that thecontrol unit performs the notification using sound.

The disclosed projection type display device is configured so that thenotification process performed by the control unit is a process ofnotifying shielding information, to be presented to the operator of thevehicle, included in information corresponding to the shielded part ofthe image light in the image information.

The disclosed projection type display device is configured so that thecontrol unit notifies the shielding information by including theshielding information in information corresponding to a part of theimage light excluding the shielded part of the image light in the imageinformation.

The disclosed projection type display device is configured so that thecontrol unit notifies the shielding information using sound.

The disclosed projection type display device is configured so that thecontrol unit notifies the shielding information by displaying theshielding information on a display device mounted in the vehicle.

The disclosed projection type display device is configured so that theprojection surface is a windshield of the vehicle, the projection typedisplay device further comprises an external object detection unit thatdetects the position of an external object in front of the projectionsurface, and the control unit determines whether to perform thenotification process on the basis of a range of the projection surfacewhere the shielded part of the image light is to be projected and theposition of the external object on the projection surface detected bythe external object detection unit.

The disclosed projection type display device is configured so that thecontrol unit prohibits the notification process in a case where theposition of the external object detected by the external objectdetection unit is outside the range of the projection surface where theshielded part of the image light is to be projected.

The disclosed projection type display device is configured so that thevehicle is a working machine.

The disclosed projection type display device is configured so that theprojection display unit is disposed at a position distant from theprojection surface with reference to an operator's seat in an indoorspace of the vehicle.

A disclosed projection control method projects image light spatiallymodulated by a light modulation element that spatially modulates lightemitted from a light source on the basis of image information onto aprojection surface mounted in a vehicle, and includes: a detection stepof detecting an image light shielded state where at least a part of theimage light is shielded by part of a body of an operator of the vehicle;and a control step of performing a notification process in a case wherethe image light shielded state is detected in the detection step.

The disclosed projection control method is configured so that thenotification process is a process of notifying the presence of the imagelight shielded state.

The disclosed projection control method is configured so that thecontrol step includes notifying the presence of the image light shieldedstate by changing information corresponding to a part of the image lightexcluding the shielded part of the image light to information includingnotification information indicating the presence of the image lightshielded state, in the image information.

The disclosed projection control method is configured so that thecontrol step includes performing the notification using sound.

The disclosed projection control method is configured so that thenotification process is a process of notifying shielding information,for performing information presentation to the operator of the vehicle,included in information corresponding to the shielded part of the imagelight in the image information.

The disclosed projection control method is configured so that thecontrol step includes notifying the shielding information by includingthe shielding information in information corresponding to a part of theimage light excluding the shielded part of the image light in the imageinformation.

The disclosed projection control method is configured so that thecontrol step includes notifying the shielding information using sound.

The disclosed projection control method is configured so that thecontrol step includes notifying the shielding information by displayingthe shielding information on a display device mounted in the vehicle.

The disclosed projection control method is configured so that theprojection surface is a windshield of the vehicle, the projectioncontrol method further comprises an external object detection step ofdetecting the position of an external object in front of the projectionsurface, and the control step includes determining whether to performthe notification process on the basis of a range of the projectionsurface where the shielded part of the image light is to be projectedand the position of the external object on the projection surfacedetected in the external object detection step.

The disclosed projection control method is configured so that thecontrol step includes prohibiting the notification process in a casewhere the position of the external object detected in the externalobject detection step is outside the range of the projection surfacewhere the shielded part of the image light is to be projected.

The disclosed projection control method is configured so that thevehicle is a working machine.

The invention is applied to a working machine such as a constructionmachine or an agricultural machine, which provides high comfort andeffectiveness.

EXPLANATION OF REFERENCES

2, 3, 4: unit

5: windshield

8: display device

9: sound output device

10: HUD

40: light source unit

45: driving unit

60, 61, 62: system controller

70, 70A: main controller

72: detection unit

73: notification processing unit

74: external object detection unit

100: construction machine

What is claimed is:
 1. A projection type display device comprising: aprojection display unit that projects image light, which is spatiallymodulated by a light modulation element that spatially modulates lightemitted from a light source on the basis of image information, onto aprojection surface mounted in a vehicle; a detection unit that iscapable of detecting an image light shielded state where at least a partof the image light is shielded by part of a body of an operator of thevehicle; and a control unit that performs a notification process in acase where the image light shielded state is detected by the detectionunit.
 2. The projection type display device according to claim 1,wherein the notification process performed by the control unit is aprocess of notifying the presence of the image light shielded state. 3.The projection type display device according to claim 2, wherein thecontrol unit notifies the presence of the image light shielded state bychanging information corresponding to a part of the image lightexcluding the shielded part of the image light to information includingnotification information indicating the presence of the image lightshielded state, in the image information.
 4. The projection type displaydevice according to claim 2, wherein the control unit performs thenotification using sound.
 5. The projection type display deviceaccording to claim 1, wherein the notification process performed by thecontrol unit is a process of notifying shielding information, to bepresented to the operator of the vehicle, included in informationcorresponding to the shielded part of the image light in the imageinformation.
 6. The projection type display device according to claim 5,wherein the control unit notifies the shielding information by includingthe shielding information in information corresponding to a part of theimage light excluding the shielded part of the image light in the imageinformation.
 7. The projection type display device according to claim 5,wherein the control unit notifies the shielding information using sound.8. The projection type display device according to claim 5, wherein thecontrol unit notifies the shielding information by displaying theshielding information on a display device mounted in the vehicle.
 9. Theprojection type display device according to claim 1, wherein theprojection surface is a windshield of the vehicle, the projection typedisplay device further comprises an external object detection unit thatdetects the position of an external object in front of the projectionsurface, and the control unit determines whether to perform thenotification process on the basis of a range of the projection surfacewhere the shielded part of the image light is to be projected and theposition of the external object on the projection surface detected bythe external object detection unit.
 10. The projection type displaydevice according to claim 9, wherein the control unit prohibits thenotification process in a case where the position of the external objectdetected by the external object detection unit is outside the range ofthe projection surface where the shielded part of the image light is tobe projected.
 11. The projection type display device according to claim1, wherein the vehicle is a working machine.
 12. The projection typedisplay device according to claim 1, wherein the projection display unitis disposed at a position distant from the projection surface withreference to an operator's seat in an indoor space of the vehicle.
 13. Aprojection control method of the projection type display deviceaccording to claim 1 for projecting image light spatially modulated bythe light modulation element that spatially modulates light emitted fromthe light source on the basis of image information onto the projectionsurface mounted in a vehicle, comprising: a detection step of detectingthe image light shielded state; and a control step of performing anotification process in a case where the image light shielded state isdetected in the detection step.
 14. The projection control methodaccording to claim 13, wherein the notification process is a process ofnotifying the presence of the image light shielded state.
 15. Theprojection control method according to claim 14, wherein the controlstep includes notifying the presence of the image light shielded stateby changing information corresponding to a part of the image lightexcluding the shielded part of the image light to information includingnotification information indicating the presence of the image lightshielded state, in the image information.
 16. The projection controlmethod according to claim 14, wherein the control step includesperforming the notification using sound.
 17. The projection controlmethod according to claim 13, wherein the notification process is aprocess of notifying shielding information, for performing informationpresentation to the operator of the vehicle, included in informationcorresponding to the shielded part of the image light in the imageinformation.
 18. The projection control method according to claim 17,wherein the control step includes notifying the shielding information byincluding the shielding information in information corresponding to apart of the image light excluding the shielded part of the image lightin the image information.
 19. The projection control method according toclaim 17, wherein the control step includes notifying the shieldinginformation using sound.
 20. The projection control method according toclaim 17, wherein the control step includes notifying the shieldinginformation by displaying the shielding information on a display devicemounted in the vehicle.
 21. The projection control method according toclaim 13, wherein the projection surface is a windshield of the vehicle,the projection control method further comprises an external objectdetection step of detecting the position of an external object in frontof the projection surface, and the control step includes determiningwhether to perform the notification process on the basis of a range ofthe projection surface where the shielded part of the image light is tobe projected and the position of the external object on the projectionsurface detected in the external object detection step.
 22. Theprojection control method according to claim 21, wherein the controlstep includes prohibiting the notification process in a case where theposition of the external object detected in the external objectdetection step is outside the range of the projection surface where theshielded part of the image light is to be projected.
 23. The projectioncontrol method according to claim 13, wherein the vehicle is a workingmachine.